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DETAILED ACTION 



1. Claims 1 - 19 are pending. 

2. The rejection of claim 10 under 35 U.S.C. 101 as being directed to non-statutory 
subject matter has been withdrawn as necessitated by amendment. 

Declaration Under 35 USC § 131 

3. The evidence submitted is insufficient to establish diligence from a date prior to 
the date of reduction to practice of the "MediaNet: A multimedia Information Network for 
Knowledge Representation", In Conference on Internet Multimedia Management 
Systems. Vol. 4210 (numbered Pages 1-12), Boston, MA, Nov. 2000. 1 ST/SPIE.OO, 
Author: Ana B. Benitez et al" reference to either a constructive reduction to practice or 
an actual reduction to practice. 

4. The following is a quotation of 35 U.S.C. 715.07(x) which forms the basis for 
diligence set forth in this Office action: 



What is meant by diligence is brought out in Christie v. Seybold, 1893 CD. 515, 64 O.G. 1650 (6th 
Cir. 1893). In patent law, an inventor is either diligent at a given time or he is not diligent; there are 
no degrees of diligence. An applicant may be diligent within the meaning of the patent law when he 
or she is doing nothing, if his or her lack of activity is excused. Note, however, that the record must 
set forth an explanation or excuse for the inactivity; the USPTO or courts will not speculate on 
possible explanations for delay or inactivity. See In re N'elson, 420 F.2d 1 079, 164 USPQ 458 
(CCPA 1970). Diligence must be judged on the basis of the particular facts in each case. See MPEP 
§ 2138.06 for a detailed discussion of the diligence requirement for proving prior invention. 



5. A general allegation that the invention was completed prior to the date of the 
reference is not sufficient. Ex parte Saunders. ; 1883 CD. 23, 23 O.G. 1224 (Comm'r 
Pat. 1883), Similarly, a declaration by the inventor to the effect that his or her invention 
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was conceived or reduced to practice prior to the reference data, without a statennent of 
facts demonstrating the correctness of the conclusion, is insufficient to satisfy 37 CFR 
1.131. 

While conception is the mental part of the inventive act, it must be capable of proof, 
such as by demonstrative evidence or by a complete disclosure to another, Conception 
is more than a vague idea of how to solve a problem. The requisite means themselves 
and their interaction must also be comprehended. See Mergenthaler v, Scudder, 1897 
CD. 724, 81 O.G. 1417 (D.C. Cir. 1897). 

6. At page 2. paragraph 2 of the affidavit, the claimant states: 

"Prior to November 6, 2000, we have conceived the claimed invention of the 
application. A Provisional Patent Application Serial No. 60/246,052 was filed on, 
November 6, 2000, with the USPTO. John R. Smith is the common inventor to the 
present application and the provisional '052 application. The '052 application as filed is 
attached herewith Exhibit A." 

7. The evidence submitted is insufficient to establish applicant's alleged actual 
reduction to practice of the invention in this country or a NAFTA or WTO member 
country after the effective date of the Benitez reference. 

The following is a quotation of 35 U.S.C. 715.07 which forms the basis for the 
explanation of the affidavit set forth in this Office action: 

The affidavit or declaration and exhibits must clearly explain which facts or data applicant is relying on to 
show completion of his or her invention prior to the particular date. Vague and general statements in broad terms 
about what the exhibits describe along with a general assertion that the exhibits describe a reduction to practice 
"amounts essentially to mere pleading, unsupported by proof or a showing of facts" and, thus, does not satisfy the 
requirements of 37 CFR 1.131(b). In re Borkowski, 505 F.2d 713, 184 USPQ 29 (CCPA 1974). Applicant must give a 
clear explanation of the exhibits pointing out exactly what facts are established and relied on by applicant. 505 F.2d at 
718-19, 184 USPQ at 33. See also In re Many, 333 F.2d 920, 142 USPC 164 (CCPA 1964) (Affidavit "asserts that 
facts exist but does not tell what they are or when they occun-ed."). 
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8. There was no indication as to which portions of Exhibit A of the affidavit are 
related to the various phrases in independent claims 1,10 and 1 1 of the patent 
application. Particularly, the elements of claim 10 are not in Exhibit A. The applicant 
should provide a correlation on how various statements in the affidavit apply to these 
claims. The display of examples of commands does not in itself provide a correlation 
between the display and elements of the claims in the proposed invention. 

For these reasons, the "Affidavit Under 37 C. F. R. § 1.131" has been disallowed. 

Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 
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10. Claims 1 and 1 1 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Non-Patent Document: "MediaNet: A multimedia Information Network for 
Knowledge Representation", In Conference on Internet Multimedia Management 
Systems. Vol. 4210 (numbered Pages 1-12), Boston, MA, Nov. 2000, 1 
ST/SPIE.OO, Author: Benitez et al (hereinafter "Benitez") in view of Non-Patent 
Document: "Supporting Ranked Boolean Similarity Queries in MARS", IEEE 
Trans, on Knowledge and date Engineering, 10, Nov - Dec. 1998, Author: 
Ortaga et al (hereinafter "Ortega"). 
Regarding claims 1 and 1 1 , Benitez teaches a method for querying stored 
multimedia data in a computer system, comprising: 

translating, in said intermediate level, said high-level concept into low-level queries 
by using system pre-defined high-level concepts (see section 3 "Typical content-based. 
In the database or are used in the retrieval" and section 4.2 page 8 "The extended 

content-based the procedure is shown in Figure 1"); and 

transferring said low-level queries to a low level comprising one or more search 
engines; said one or more search engines performing a query of the stored multimedia 

information using said low-level queries (see section 4.2, page 9, "At this point and 

the results integrated into a unique list as described for visual queries"). 

Benitez does not explicitly teach receiving into an intermediate level a high-level 
concept from a user describing data to be retrieved. 
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Ortega teaches receiving into an intermediate level a high-level concept from a user 
describing data to be retrieved (see section 1 .2, page 4, and paragraph 1 , "A Boolean 
... image databases"). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine teaching of Ortega with the teaching of Benitez wherein high 
level concepts can be expressed as a Boolean combination of lower level features. The 
motivation is that the user for the support of conceptual queries can provide such 
mapping of high to low level concepts explicitly. 

H.CIaims 2- 10, and 12 - 19 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Benitez in view of Ortaga and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated image Libraries", In Storage and 
Retrieval for Image and Video Database, San Jose, CA, Jan. 2001 . SPIE 
(numbered pages 1-12), Author: Natsev et al (hereinafter "Natsev"). 

Regarding claims 2 and 12, Benitez and Ortega disclose the claimed subject 
matter as discussed in claims 1 and 1 1 respectively. Benitez or Ortega does not 
explicitly teach wherein said intermediate level comprises: a set of library modules, 
said set of library modules comprising: a concept library module for storing concepts; 

one or more library modules adapted to store said data from said one or more 
data sources; a cataloger module adapted to construct a new concept from said 
high-level concept using data from said concept library and library modules, thereby 
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creating a concept construct, and to pass said concept construct to said concept 
library module for storage as a concept; and an interpreter module adapted to 
translate said high-level concept into low-level queries using said concepts stored in 
said construct library and to pass said low-level queries to said one or more search 
engines, 

Netsev teaches wherein said intermediate level comprises: a set of library 
modules, said set of library modules comprising: 

a concept library module for storing concepts (see abstract, page 1); 

one or more library modules adapted to store said data from said one or more 
data sources (see figure 1 , section 1 ,2 paragraph 3, page 3 - page 4, paragraph 1 , 
"The concept Library is a module for persistent storage of concepts. ...terminology"); 

a cataloger module adapted to construct a new concept from said high-level 
concept using data from said concept library and library modules, thereby creating a 
concept construct, and to pass said concept construct to said concept library module 
for storage as a concept (see fig.3, section 1 .2, page 3); and 

an interpreter module adapted to translate said high-level concept into low-level 
queries using said concepts stored in said construct library and to pass said low- 
level queries to said one or more search engines (see section 1 .2 page 4 "The 
querying phase simply. ...keyword query"). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Natsev with the teaching of Benitez and 
Ortega wherein the concept interpreter looks up the representation of the visual 
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concept from the concept library and uses it to query the database for images 
containing the relevant concept. The motivation is that once a concept is defined, it 
can be used to search for relevant images without having annotated each and every 
one of them. 

Regarding claims 3 and 13, Benitez, Ortega and Natsev disclose the claimed subject 
matter as discussed in claims 2 and 12 respectively. Ortega teaches wherein said set of 
library modules further comprises at least one library module selected from the group 
comprising: a matching algorithm library module adapted to store matching algorithms 
(see section 4.8, pages 20 and 21). 

Benitez or Ortega does not explicitly teach a feature library module adapted to store 
multimedia features; and a constraint library module adapted to store feature 
constraints. 

Natsev teaches a feature library module adapted to store multimedia features (see 
section 1 .2, paragraph 2, page 3); and 

a constraint library module adapted to store feature constraints (see section 5, 
pages 7 and 8). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Natsev with the teaching of Benitez and 
Ortega wherein the concept interpreter looks up the representation of the visual concept 
from the concept library and uses it to query the database for images containing the 
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relevant concept. The motivation is that once a concept is defined, it can be used to 
search for relevant images without having annotated each and every one of them. 

Regarding claims 4 and 14, Natsev teaches wherein each said library module further 
comprises an application program interface to receive said data from a said data source 
(see section 4, page 7, "The last piece ... .in Figures 4 and 5."). 

Regarding claims 5 and 15, Natsev teaches wherein said cataloger module further 
performs the steps of: 

selecting a set of concept features from said feature library module (see section 1 .2, 
paragraph 2, page 3); 

selecting a set of concepts from said concept library module for use as child- 
concepts (see section 6, paragraph 3, page 1 1 ); and 

selecting a set of constraints on said child concepts from said constraint library 
module (see section 6, paragraph 3, page 11). 

Regarding claims 6 and 16, Natsev teaches wherein said each said concept 
comprises a triplet of a set of child-concepts, a set of features, and a set of relationships 
(see section 6, paragraph 3, page 1 1 , "Another improvement . . . ranked higher"). 

Regarding claims 7 and 17, Ortega teaches wherein said concepts comprise a 
hierarchical fuzzy graph data tree-structure comprising nodes, aggregation edges, and 
association edges and wherein: said nodes correspond to said concepts and said 
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features; said aggregation edges correspond to parent-child relationships; and said 
association edges correspond to said constraints (see section 4.1, pages 10 and 11). 

Regarding claims 8 and 18, Ortega teaches wherein said edges are weighted (see 
section 4.2 page 11). 

Regarding claims 9, 10 and 19, Ortega teaches a program storage device readable 
by machine, tangibly embodying a program of instructions executable by the machine to 
perform method steps for a matching algorithm, said method comprising: 
GetNextMatch( ), AssignNextMatch( ), and ShiftNextMatch( ) procedures, wherein: 
said GetNextMatch( ) procedure comprises the steps: 
testqueue: if queue.Empty( ); 
return NULL, 
head -> queue. Pop( ); 
if head.Complete( ); 
return head; head2 --> head.Copy( ); 
head2.AssignNextMatch( ); 

if head2.Valid( ); 
queue.Push(head2); 
head.ShiftNextMatch( ); 
queue.Push(head); 
Goto testqueue; 
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said AssignNextMatcli( ) procedure comprises the steps: 
child --> GetNextUnassigned( ); child.match_ptr++; 
if(child.match_ptr] == NULL), then; child.match_ptr --> 
child.GetNextMatch( ); 

IVIal^e child an assigned node; 

said ShiftNextMatch( ) procedure comprises the steps: 
Child --> GetNextUnasslgned( ); 
child. match_ptr++; 
if(child.match_ptr == NULL), then; 

child. nnatch_ptr --> child. GetNextMatch( ); 
wherein variables head, head2, and child, all correspond to concept nodes; 
variable queue denotes a priority queue of the corresponding concept node; and 
match_ptr is a pointer to the next possible match for a given concept node; Pop( ) is a 
method to get the next node off the priority queue; Push( ) is a method to put a node on 
the priority queue; Empty( ) is a method to check if the priority queue is empty; Copy( ) 
is a method to copy a node; Complete( ) is a method to check if the children assignment 
is complete; Valid( ) is a method to check if the children assignment meets the 
constraints; and GetNextUnassigned( ) is a method to select a variable that is 
unassigned (see sections 4.6 - 4.9; pages 15 - 26). 

Nonfunctional descriptive material cannot render nonobvious an invention that 
would have otherwise been obvious. Cf. In re Gulack, 703 F.2d 1381, 1385, 217 
UPPQ 401, 404, (Fed. Cir. 1983) (When descriptive material is not functionally 
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related to the substrate, the descriptive material will not distinguish the invention 
from the prior art in terms of patentability). The difference between the prior art and 
the claimed Invention Is simply a rearrangement of non-functional descriptive 
material (MPEP 2106 VI). Applicants emphatically stated in the specification that 
Ortega discloses a similar problem solving algorithms which when evaluated give 
the same result as the current invention (pages 5 and 6 of the specification). 

Therefore, It would have been obvious to one of ordinary skill In the art at the 
time the Invention was made to rearrange the algorithm of Ortega to perform the 
function of GetNextMatch( ), AssignNextMatch( ), and ShiftNextMatch( ) procedures. 



12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply Is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action Is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action Is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 



Conclusion 
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Any inquiry concerning this communication or earlier communications from ttie 
examiner should be directed to Fred I. Ehichioya whose telephone number is 703-305- 
8039. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 703-305-9790. The fax phone number 
for the organization where this application or proceeding Is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



Fred I. Ehichioya 
Examiner 
Art Unit 2172 
June 9, 2004 




